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A method for identifying closed areas and 
defining masks In a digital Image 



^ / The present invention relates to the processing of images in general, and in 
5 particular to systems for processing digital images. 

The invention has been developed with particular reference to a method for 
identifying closed areas and defining corresponding masking zones with respect to 
the contours of objects depicted in a complex image, for instance the Image of a 
1 0 building or of the interior of furnished premises. 



An increasing number of applications require the masking of specific zones of a 
digital image for subsequent processing, for instance in order to cut out specific 
images from the image, or to modify the colour or the shape of particular objects, 
15 leaving the remainder of the image unchanged, or vice versa. A widely known 
system is the "croma-key" system. This system, of analog type, makes it possible 
to remove those areas having a predetermined colour from an image and is often 
used to superimpose the image of a person on a background selected at will. 

20 The croma-key system and its drawbacks, and a system for remedying these 
drawbacks with reference to the processing of a digital image, are set out in detail 
in US Patent US-A-5 469 536. This patent discloses a method of image 
processing with a masking facility that remedies the problems of the conventional 
croma-key system and makes it possible to define, from the digital image of a 

25 person on a substantially uniform background, the contour separating the person 
and this background, so as to be able to remove those portions of the image, i.e. 
the background itself, located outside the contour. In this way, the resulting image 
can be superimposed on a background of different type previously input and 
stored in digital fomri in the memory of an electronic processor. The method by 

30 which the contour is identified consists In organising the points of the digital image 
in successive rows and analysing the colour of these points in succession until, for 
each line, there is a difference between the chromatic hue values of two adjacent 
points that is greater than a predetermined threshold. One of the two points at 
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which such a difference in chromatic hue is detected is therefore defined as 
belonging to the background and the other to the image to be masked. The 
method takes place simultaneously for the right-hand and left-hand side of the 
image. 

5 

The method disclosed in the Patent US-A-5 469 536 has the drawback that the 
background of the digital image necessarily has to be of a substantially uniform 
colour. Moreover, this system can locate only the contour of convex images, for 
instance the image of a person's face. The method is unable to locate the contour 

10 of concave objects that have zones with recesses, niches or hollows. The method 
is unable, moreover, correctly to detect the contours of a plurality of objects 
depicted in an image, when these are partially superimposed or have their 
respective contours entirely enclosed within one another. A further drawback lies 
in the fact that the method is unable to detect any holes or cut-outs in the image to 

15 be masked which, for instance, allow the background to be seen or to show 
through zones contained within the outermost contour of an object. 

Various solutions have been proposed to remedy the above-mentioned problems, 
which require varying degrees of interaction with a user, for instance roughly to 

20 define a vicinity of the image in which to search for the contour of a figure, or to 
define, in a limited zone of the image, the difference between chromatic hues 
representative of the background and the figure respectively. An example of these 
user-guided masking procedures is the masking product known by the trade name 
Mask Pro marketed by the Extensis Corporation as described as the Internet 

25 address http://www.extensis.com/. 

Although the above-mentioned systems make it possible to achieve precise 
masking results, they are intended, however, for professional use and require 
substantial experience of use, in addition to the fact that the identification of 
30 closed areas in a complex image and their masking is generally time-consuming. 

Among others, an application of particular interest of masking systems is in the 
sector of so-called tintometric systems, in which a metering machine supplies 
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predetermined quantities of one or more colouring agents concentrated within a 
container containing a base product of unifonn colour, in most cases white or 
transparent, in order to obtain a final paint of the desired colour. In this sector, it is 
increasingly necessary to be able to provide potential customers with an advance 
idea of the result that could be obtained with the final paint when it is for instance 
applied to objects or types of fumishings, or when it is used to paint the walls of a 
room or even the outside of a building. Techniques are known in this respect for 
displaying the final colour of a paint, on the basis of parameters identifying this 
paint, on the screen of a electronic processor. 

Merely displaying the final paint colour in advance is often not enough, however, 
to give an idea of the final result and it is becoming increasingly necessary to 
show potential customers what effect this paint will have when applied to a 
particular object. The object of the present invention is to provide a processing 
system for a digital image that makes it possible rapidly and simply to identify the 
contours of figures within this image, in order to be able to mask these figures 
independently from one another and from the sun-ounding background zones. A 
further object of the invention is to provide a system of the above-mentioned type 
that can work in combination with a tintometric system, and in particular with a 
colouring agent metering device, in order to provide as output a paint of a desired 
colour. The possibility of using, for the application of the system of the present 
invention, the same electronic processor adapted to control this colouring agent 
metering machine, is particularly useful in this respect. 

In order to achieve these objects, the invention relates to the method as set out in 
the accompanying claims. 

As a result of the present invention, potential customers may for instance go to a 
sales point that uses a tintometric system, with an image depicting the object, 
room or building whose colour they wish to change. If this image is not already in 
digital format, it can be converted Into this fomiat, for instance using a scanner, 
and input into an electronic processor, preferably the same processor used to 
control the paint metering machine. Once the image has been input, the system of 
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the present invention identifies the contours of closed areas within the image, 
corresponding to the contours of objects depicted therein. The custonner can then 
use an inputting system, for instance a keyboard, a mouse, a joystick or the like, 
to interact with specific figures selected and masked within this image and 
5 displayed on a video screen, or with the areas external thereto, for instance in 
order to modify the colour of the walls of a room without changing the colour of the 
fumishings visible in the image. 



\j y/ Further characteristic features and advantages of the present invention are set out 
10 in the following description of a preferred embodiment thereof, given by way of 
non-limiting example, and made with reference to the accompanying drawings, in 



Fig. 1 is an example of an image converted into grey tones for the application of 
1 5 the method of the present invention; 

Fig. 2 is a representation of the image of Fig. 1, at the end of processing by an 
edge-detection algorithm and of a possible border closure cycle, 

20 Fig. 3 is a representation of the image of Fig. 2, after removal of the dead 
branches of the borders detected; 

Fig. 4 is a block diagram of the method of the present invention; 

25 Fig. 5 shows diagrammatic examples of relationships between the point of a 
detected border and the neighbouring points or clusters of points; 

Fig. 6 shows an example of closed edges identified during the method of the 
present invention; 

30 

Fig. 7 shows an example of the stage of filling of the zones external to a closed 
edge, in order to identify a closed masking area. 




which: 
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The method of the present invention is adapted to work in combination with an 
electronic processor comprising a memory, a display unit or video screen, and an 
inputting device such as a keyboard, mouse, joystick or the like. The electronic 
processor is preferably, although in a non-limiting manner, connected to an image 
data acquisition device such as a scanner. The electronic processor is optionally 
connected to a data network or to a telephone line or the like. 

A digital image comprises a plurality of points or pixels organised in successive 
rows, each of which is defined by a group of values stored in the memory of the 
electronic processor and identifying the colour characteristics of this point. 

Known digital image acquisition devices (video cameras, digital cameras, colour 
scanners, etc.) generally encode each point of an image using a trio of values that 
expresses the level of the red. green and blue tones of the colour of the point 
(RGB code) which define the point's "colour coordinates". On the basis of this 
code, an algorithm for converting the colour coordinates of the point is applied and 
supplies as output a trio of values expressing the level of chromatic hue, 
luminance and saturation of the colour of this point (HLS code). This conversion is 
useful for the purposes of the subsequent stage of the method of the present 
invention which consists in converting a digital colour image into a digital image in 
grey tones, an example of which is shown in Fig. 1. Each point or pixel of the 
digital image 10 is associated with a data structure POINT containing the value 
POINT_VALUE which identifies only the relative luminance of the point. In the 
system of the present invention, therefore, in contrast to known systems, and 
despite the adaptation of the system for use in the colour management sector, the 
information on chromatic hue present in the original image is substantially 
disregarded or reduced solely to the level of luminance gradation. 

The luminance data of the points of the digital image, stored in the field 
POINT_VALUE. are processed by an edge-detection algorithm implemented in 
the form of software on the electronic processor. A wide range of algorithms 
adapted for this purpose are known, whose description lies outside the scope of 
the present invention, which supply as output, for each point of the digital image, a 
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value representative of the luminance gradient of the neighbouring points. For the 
purposes of this description, without this being considered limiting, it will 
conventionally be assumed that this value is the higher, the more abrupt the 
luminance variation between the point in question and its neighbouring points, i.e. 
the higher the luminance gradient. 

If the value of the luminance gradient of a certain point is high and exceeds a 
predetermined threshold value, this point is considered to be a border between 
zones of different luminance, which could be the border separating an object 
depicted in the digital image and the background or the border between two 
objects. The best edge-detection algorithms for the purposes of the present 
invention supply as output borders of a width substantially equal to a point or pixel, 
so as to define a map of substantially elongate and linear borders. 

If the value of the luminance gradient in a point exceeds the above-mentioned 
threshold, this point is conventionally considered to be "black" and, for simplicity of 
explanation, will be referred to in the following description as a "border" point. In 
contrast, those points in which the above-mentioned value of the luminance 
gradient does not exceed the above-mentioned threshold value are conventionally 
defined as "white". At the end of the edge-detection algorithm, the data stored in 
POINT_VALUE contain the values of the luminance gradients of the various 
points of the image. These values are then interpreted as "black" or "white" (or in 
an equivalent manner as "true" or "false") values, by comparing them with the 
above-mentioned threshold value. 

These values of white (false) and black (true) points are successively considered 
during a closure cycle, repeated one or more times, and in turn comprising a first 
stage of edge labelling (LabelEdges), a second stage of scanning to close 
(ScanToClose) and a third stage of definition of the neighbouring points 
(SetRealNeighbours). In more detail, during the stage of edge labelling, each point 
is considered on the basis of the intensity of the luminance gradient detected, in 
particular on the basis of whether it is black or white, in practice by reading the 
value of the datum POINT_VALUE. During this stage, a vector of data structures 
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BORDER is generated, and is stored on the processor separately from the vector 
of structures POINT of the point data, in which those borders that show continuity, 
i.e. that have one or nnore neighbours, wherein a neighbour is understood as an 
adjacent "black" point or cluster of "black" points, are progressively listed. Each 
5 border is identified by a label BORDER_LABEL, in practice a progressive 
numerical value, and a length value BORDER_LENGTH. equal to the number of 
points or pixels from which the linear development of this border is composed, is 
assigned thereto. In a variant of the present invention, each border is also 
assigned a coherence value BORDER_COHERENCE, indicative of the regularity 
10 of the border which will, for instance, be the higher, the more the border is regular, 
and the lower, the more the border is disjointed or irregular. During the labelling 
stage LabelEdges, all the borders, whether closed or open, and of whatever 
length, are in practice identified, provided that at least one point of the border has 
a sufficiently significant gradient, i.e. above the predetermined threshold value. 

15 

Each point of the image is thus associated, in addition to the value 
POINT^VALUE. with a further label value POINT_LABEL which can assume the 
values I_ABEL_NONE (when the point is white and does not belong to any 
identified border), LABEL_BOUNDARY (when the point is located on the edge of 
20 the digital image), or a label value corresponding to the progressive identification 
list of the corresponding border as identified by the value BORDER_LABEL in the 
manner described above. 

In the subsequent stage of scanning to close, it is attempted to close the 
25 incomplete borders to generate closed edges, by adding "black" points to the map 
of points processed by the edge-detection algorithm. Fig. 2 shows an example of 
the image obtained from the image 10 of Fig. 1. displaying the borders obtained at 
the end of the stage of scanning to close. In particular, use is made for this 
purpose of the border length value BORDER_LENGTH identified in the preceding 
30 labelling stage LabelEdge. A fairly rapid method consists in seeking to prolong a 
border only if its value BORDER_LENGTH is sufficiently long and higher than a 
threshold value that can be predetermined or detemnined on the basis of the mean 
of the length of borders identified and stored in the structures BORDER. Starting 
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from an open end of one these open borders, a search is carried out among the 
surrounding points to identify a further black point, and thus to prolong the open 
end by adding intermediate black points. When considering these surrounding 
black points, points belonging to the same border to which the same end point in 
5 question also belongs are disregarded, in order to avoid closing the border on 
itself. 

The distance in points or pixels to which reference is made when searching for 
another border from the end of an open border is proportional to the value of the 

10 length of the latter (BORDER^LENGTH), so as to avoid prolonging very short and 
often incidental fragments and to promote the closure of borders whose linear 
extension is of particular significance. During this stage, the choice of the path of 
closure of the open border, i.e. of the sequence of "white" points to be converted 
into "black" points, can be optimised in various ways. For instance, account could 

15 initially be taken of all the points that lie on the most direct trajectory or on the 
shortest path that Joins the end of the open border to the target border. As an 
alternative, or in combination, account can be taken of the actual value of the 
luminance gradient of the adjacent points, even though these have initially been 
considered as "white" points. This means that the various possible trajectories for 

20 joining the end of the open border with the target border include those containing 
points whose luminance gradient is slightly below the threshold separating the 
black points from the white points, but nevertheless greater than that of all the 
other surrounding points not belonging to the open border. For practical purposes, 
it is enough temporarily and locally to lower the threshold value of the luminance 

25 gradient in order to cause any further "black" points that can be used as an 
indication of the best path to be followed to close the open border to "emerge" 
from the cluster of "white" points adjacent to the end of the open border. In 
addition or as an alternative to the criteria for choosing the above-described 
closure path. account can be taken of the coherence value 

30 BORDER_COHERENCE in order to identify a border closure path that is most in 
keeping with the morphological characteristics of this border. 

At the end of each repetition of the stage of scanning to close CloseToScan, the 
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Stage of definition of the neiglibouring points SetRealNeiglibours defines (when 
first run) or updates (when subsequently run) a numerical value 
POINT_NEIGHBOURS representative of the number of clusters of black points 
bordering each black point P of the digital image. In particular, with reference to 
Fig. 5, this numerical value is: 

0: when the black point P in question has no neighbouring black point and is 
therefore an isolated point in the map M of the Image, not belonging to any border 
(example 5a); 

1: when the black point P in question is the end point of a branch of open 
border (example 5b); 

2: when the black point P in question is a point belonging to a generic border 
(example 5c); 

3 or 4: when the black point P in question is the point of intersection of several 
branches of a border (example 5d). 

Following the stage of re-definition of the neighbouring points SetRealNeighbours, 
the data BORDER_U\BEL and BORDER_LENGTH are updated, making further 
use of the above-mentioned stage of labelling of the borders LabelEdges. to 
reflect the fact that some borders have been combined to fonn closed edges in 
the preceding stage of scanning to close ScanToClose. 

The closure cycle comprising the three stages described above is repeated until it 
is no longer possible to close any edges. In practice, a maximum number of 
possible repetitions is set in order to prevent any risk of overly long processing 
times. Experimentally, it has been found that a maximum of twenty or so 
repetitions provides very acceptable closure results. 

At the end of the closure cycle, the digital image has a series of closed edges and 
open branches, the latter being fomned by the residual open borders that were too 
short or too distant from other borders to be closed in the closure cycle. Fig. 2 
shows an example of the graphical representation of the resulting digital image. A 
successive stage of the system of the present invention consists in the pruning of 
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the open branches. For this purpose, account is taken of the ends of the borders 
or branches that have remained open despite the application of the closure cycle, 
and in particular of the black points for which the value POINT_NEIGHBOURS Is 
equal to unity. From the end of each open branch and proceeding backwards 
5 along this branch, pruning takes place by decreasing the value 
POINT_NEIGHBOURS of each black point of the open branch by one unit, until 
an Intersection, for which the value of POINT_NEIGHBOURS, although decreased 
by one unit, still remains above one, is encountered. In practice, the starting point 
is the end of an open or dead branch where the value of POINT_NEIGHBOURS 

10 decreases from one to zero. The adjacent black point is then evaluated, 
decreasing the value of POINT_NEIGHBOURS by one unit for this point as well. 
There are two possibilities at this point: if the new value of POINT_NEIGHBOURS 
is equal to unity, then this black point Is the new end of the open branch and Its 
corresponding adjacent black point is then considered, or the new value of 

15 POINT_NEIGHBOURS is greater than unity and the pruning procedure is then 
discontinued because an Intersection has been encountered. 

At the end of a pruning operation, all the points of a dead branch will have a 
POINT_NEIGHBOURS value equal to zero, apart from the "node" point. I.e. the 
20 black point located at the intersection of the dead branch with another branch or 
edge. In practice, at the end of the pruning operation, all the points of the digital 
image belonging to closed edges will have a POINT_NEIGHBOURS value equal 
to or greater than two. For the successive stages of the method it is sufficient 
therefore to take account solely of the POINT_NEIGHBOURS value of each point. 

25 

The method described up to now is intended to identify closed edges of non- 
negligible length and to label them, making It possible to start from the map of 
borders detected by the edge-detection algorithm (Fig. 2) in order to close any 
edges that are open as a result of small discontinuities that cannot be detected 
30 from the original digital image (Fig. 1), eliminating the open branches that have not 
been closed (Fig. 3). 

Some areas, albeit only edges closed one or more times, have not. however, been 
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identified. As shown in Fig. 6, the method up to this point has labelled as closed 
edges both simple edges 6a and the edges 6b and 6c which are also closed but 
more complex. This latter type of closed edge 6c is defined as a "lens edge" and 
comprises a border whose ends are closed on themselves to form two "lenses" or 
5 eyes, connected together by the intermediate section of the border. These lens 
edges are problematic and often impede the elimination of borders that do not 
define closed edges from the data of an image. It is therefore preferable to carry 
out a stage of elimination of lens edges in order to try to remove spurious 
elements from the digital image data. 



The stage of elimination of the lens edges is in practice indispensable when, for 
the construction of masks, use is made of known methods for the construction of 
polygons from ordered series of points. By definition, these polygons are bounded 
by closed edges. In this case, unless the edges are completely eliminated, the 
15 results of the final masking will be unpredictable and incorrect. In the preferred 
embodiment of the invention, however, as specified in detail below, it is not strictly 
necessary to remove the lens edges, since the masking of the digital image can 
be carried out by means of a stage of definition of simple closed areas, without the 
need to order the points of the closed edges that bound them. 



In the stage of elimination of the lens edges it is possible to prune the intermediate 
section that unites the two end eyes, or to assimilate each end eye with the end of 
an open border, which is closed in a manner substantially similar to that described 
with reference to the cycle of closure of open borders. The choice of the type of 

25 method to be used to eliminate the lens edges depends to a large extent on the 
length of the intermediate section related to the length of the end eyes. 
Advantageously, the stage of elimination of the lens edges may be included within 
the closure cycle as described above, prior to the pruning stage, so as also to take 
account of the borders that have a single end eye. of small dimension, that can be 

30 assimilated with the end of an open border. 

At the end of the cycle of closure, pruning and possible elimination of lens edges, 
the masking data are collected in the form of an object vector, each of which 
contains the data relating to an individual closed edge that has been detected, in 



10 



20 
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particular the value of its length BORDER_LENGTH and the list of the points 
belonging thereto, on the basis of the labelling of these points by means of the 
value POINT_LABEL. The digital image is scanned according to a sequence from 
the top-left to the bottom-right In order always securely to find, in the succession of 
closed edges, an outer edge (container) before a corresponding edge contained 
therein. This method consists in listing the various closed edges in an ordered 
sequence. The individual points belonging to the various edges are not collected, 
however, in any particular order, since this not essential for the purposes of the 
present invention and makes it possible to speed up the masking method. 

Subsequently, the points of each edge, taken in an ordered manner from the 
ordered sequence of edges obtained from the top-left to bottom-right scanning as 
described above, are added individually to a temporary auxiliary map. In this map, 
each point or pixel is coloured, for instance in white or black, or in two other 
distinct colours, depending on the "black" or "white" value of the value 
POINT_VALUE detemnined in the previous stages. A successive operation to fill 
the zone extemal to the edges with a colour different from the first two colours 
makes it possible immediately to identify the area inside and the area outside the 
edge which defines the broadest perimeter of the figure. This stage is shown in 
diagram form in Fig. 7, in which 7a shows an auxiliary map M1 to which the points 
of a closed edge B have been added. Fig. 7b shows a an intemiediate stage of 
the operation to fill the zone E extemal to the closed edge B, and Fig. 7c shows 
the resulting masking area K identified by the filling operation. The filling stage is 
repeated for each closed edge identified in the previous stages, thereby identifying 
a sequence of closed areas. 

The individual closed areas identified on the temporary map are input into a 
common map. in their original position, with reference to the position data of the 
points of the original digital image. The areas are input into the common map in 
the order defined by the ordered sequence of edges identified by the top-left to 
bottom-right scanning, so as to be able simply to verify whether any one of their 
points matches, i.e. coincides with, an internal point of an area defined by a 
previous edge in the above-mentioned sequential order. It is thus possible simply 
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to verify whether each area is or is not contained within other closed areas 
previously input into the common map, thereby making it possible to organise 
these areas according to a hierarchy of container-contained areas, for instance to 
manage any holes in the masks. For instance, the outennost areas can be 
considered to be full, while the successive areas in the container-contained 
hierarchy, the so-called "daughter" areas, can be considered as holes in the areas 
higher up the hierarchy, which are thus "emptied" in the zone occupied by the 
daughter area that they contain. In the example of Fig. 3, the closed edge 13 
defines the inner area of the frame 1 1 of the window shown in the image of Fig. 1 , 
while the closed edges 14 define holes in this frame 11, con-esponding to the 
glass panes 12 of the window of Fig. 1. The method of the present invention thus 
makes it possible to achieve the objective of defining In a simple and rapid 
manner, without intervention by the user, a mask for the object formed by the 
frame 1 1 of the window depicted in the image of Fig. 1 . 

At this point, the user may for instance select a colour at will from the database 
stored on the electronic processor and apply this colour to the mask defining the 
frame 1 1 in order to change, on the original colour image, only the chromatic hue 
of this frame. Moreover, the luminance data of the points of the original image are 
still available, so that the luminance parameter of the new colour selected by the 
user may be graduated, point by point, within the mask of the frame 1 1 , in order to 
obtain a very realistic colouring effect of the digital image that respects the light 
and shade contrasts of the real subject as originally photographed. 

The above-described procedure, which is fully automatic, may be integrated with a 
stage of interactive modification guided by the user. Depending on the 
characteristics of the original image, for instance, the border closure cycle could 
have generated closed edges surplus to actual requirements, for instance the 
closed edge 15 of Fig. 3. It is sufficient to provide a device for modifying the 
borders that works on the map of borders of Fig. 3, guided by the inputting device, 
for instance a mouse. By using this device of an "eraser" type it is possible to 
cancel some points of the closed edge 15 to convert it into an open border. 
Subsequent transition through the automatic pruning stage will eliminate the edge 
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15 which the user has changed into an open border. 



In contrast, the border map of Fig. 2, obtained at the end of the closure cycle prior 
to the pruning stage may be displayed to a user who wishes to close any borders 
5 that are still open. For this purpose use can be made of a device of the "stylus" 
type which makes it possible to draw lines of a width equal to a point or a pixel, 
blackening the points indicated by the user on the map by means of an interactive 
inputting device such as a mouse. 

10 The stage of filling and definition of the hierarchy of closed areas may also be 
made interactive with a user. As shown In Fig. 7. the automatic filling procedure 
considers only the closed area surrounded by the most peripheral profile of an 
edge B closed several times. In the final mask K, the sub-areas K1 , K2, K3 and 
K4 of example 7b are ignored. The masking map, for instance that shown by 7b in 

15 Fig. 7, may be displayed to a user who, by means of an inputting device, may 
indicate one of the points inside one of the sub-areas K1, K2, K3, K4 in order to fill 
its interior, aggregate it with the external area E and remove it from the closed 
masking area K. 

20 Without prejudice to the principle of the invention, its embodiments and details of 
construction may be widely varied with respect to those described and illustrated 
without thereby departing from the scope of the present invention. 



